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■  DOD  desires  more  credible  estimates 

■  Provide  for  more  realistic  Program  Element  budgets 

■  Permit  better  assessment  of  Contractor  proposed  costs 

■  Reduce  bad  press  and  the  perception  that  it  is  wasteful  of 
taxpayer  dollars 

■  To  achieve  this  goal,  DOD  needs 

■  Benchmarks  that  can  be  used  to  assess  reasonableness  of 
budget  submissions  and  Contractor  proposals 

■  Consistent  ways  of  addressing  the  many  different  size  and 
cost  parameters  used  in  preparing  estimates  and  cost-to- 
complete  exercises 

■  Hard  data  that  is  statistically  solid  that  justifies  findings 

■  With  this  data,  improvements  can  be  made  in 
processes  used  for  planning,  budgeting  and  control 
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Data  Quality  Challenges 
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■  DoD  has  many  software  cost  data  repositories 

■  DCARC,  NRO,  AFCAA,  ODASA-CE,  NCCA,  NS  A,  Aerospace,  ... 

■  Despite  increased  interest  in  data,  it  is  surprising  that  so  little 
effort  has  been  committed  reconciling  inconsistencies  within 
and  across  data  repositories... 

1 .  No  reporting  of  Equivalent  Size  Inputs  -  CM,  DM,  IM,  SU,  UNFM,  Type 

2.  No  common  SLOC  reporting  -  logical,  physical,  etc. 

3.  No  standard  definitions  -  Application  Domain,  Build,  Increment,  Spiral,... 

4.  No  common  effort  reporting  -  analysis,  design,  code,  test,  CM,  QA,... 

5.  Product  size  only  reported  in  lines  of  code 

6.  No  reporting  of  quality  measures  -  defect  density,  defect  containment,  etc. 

■  The  data  needs  to  be  statistically  sound  and  defendable 

■  A  good  opportunity  to  communicate  and  work  towards 
standardization  is  the  current  MIL-STD  881  efforts 

The  key  to  standardization  is  encouraging  broader  participation  among  these  groups 
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Limited  Research  within  DoD  \  ] \ 


1.  Other  contributors  to  Productivity  besides  effort  and 
size,  are  being  ignored  by  most  analysts 

■  Operating  Environment,  Application  Domain,  and  Product  Complexity 

■  Personnel  Capability 

■  Required  Reliability 

■  Quality  -  Defect  Density,  Defect  Containment 

■  Integrating  code  from  previous  deliveries  -  Builds,  Spirals,  Increments,  etc. 

■  Requirements  Volatility 

2.  Converting  to  Equivalent  SLOC 

■  Categories  like  Modified,  Reused,  Adopted,  Managed,  and  Used  add  no  value 
unless  they  translate  into  single  or  unique  narrow  ranges  of  DM,  CM,  and  IM 

parameter  values.  We  have  seen  no  empirical  evidence  that  they  do 

■  Other  categories  like  COTS,  Converted,  Generated  and  Rehosted  are  handled 
differently  and  there  is  no  consistency  when  they’re  used 
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Model  Calibration  Challenges  [  HJ  | 


■  Most  program  offices  and  support  contractors  rely  heavily 
on  software  cost  models  for  their  estimates 

■  May  have  not  been  calibrated  with  most  recent  DoD  data 

■  Calibration  with  recent  data  (2002-Present)  will  help 
increase  program  office  estimating  accuracy 

SEER 

SLIM-Estimate™ 

TruePlanning®  by  PRICE  Systems 
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Consequence: 
Significant  Cost  Growth  (%) 


Statistics 

*Total  System 

**Software  Only 

Minimum 

-64% 

-80% 

Mean 

45% 

37% 

Maximum 

471% 

623% 

Standard  Deviation 

71% 

107% 

Milestone  Phase 

Development 

Development 

Sample  Size 

137 

111 

Year  of  Data 

1993-2003 

2002-2008 

Source  :  *John  McCrillis,  36th  DOD  Cost  Analysis  Symposium  (2003) 
“Defense  Automated  Cost  Information  System  (DACIMS) 
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PROPOSED  SOLUTION 
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AFCAA  in  conjunction  with  USC  and  other  DOD  Cost 
Agencies,  will  publish  a  Manual  to  help  analysts  develop 
more  credible  estimates  based  on  empirical  data  in  a 
timely  and  consistent  manner 
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Special  Features  \  |^|  \ 


■  Augment  NCCA/AFCAA  Software  Cost  Handbook: 

■  Empirical  Productivity  Benchmarks  by  Operating  Environment, 
Application  Domain,  and  Software  Size 

■  Default  Equivalent  SLOC  Inputs  (DM,  CM,  IM,  SU,  UNFM); 
Requirements  Volatility  derived  from  empirical  data 

■  Empirical  Code,  Effort,  and  Schedule  Growth  Measures  derived 
from  SRDRs 

■  Guidelines  and  Knowledge  Base  that  capture  best  practice 

■  Empirical  Cost  Risk  and  Uncertainty  Analysis  Metrics 

■  Calibrated  SLIM-Estimate™  using  most  recent  DoD  data 

■  Mapping  between  COCOMO,  SEER,  True  S  cost  drivers 

■  Empirical  Dataset  for  COCOMO,  True  S,  and  SEER  Calibration 

■  Software  Maintenance  Cost  Model 
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Our  Plan  of  Attack 
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Step  1:  Data  Definition  Approach  \  | 

1.  Review  literature/ 
past  research  results 

1  Lots  of  really  good  work  already  done  by  \ 

|  variety  of  sources  \ 

2.  Update  USC’s  cost 
model  comparisons 

\  Model  comparisons  done  for  NASA  last  | 

|  year  I 

3.  Synthesize  Overall 
Framework 

1  Develop  framework  that  builds  on  past  ! 

■  to  achieve  goals  established  for  effort  ! 

4.  Identify  Candidate 
Application  Domains 

i  Estimate  accuracy  have  been  shown  to  be  \ 

1  better  when  they  are  domain-specific  \ 

5.  Define  Counting 

Rules  and  Standards 

1  Use  framework  as  basis  to  relate  counting  | 

1  rules  and  standards  1 

6.  Validate  Framework 

Via  Trial  Use 

1  Make  sure  it  works  with  real  data;  else,  ; 

!  change  it.  J 

1 — ►  Software  Cost  Handbook 
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Operating  Environments: 
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Operating  Environments 


1 .  Avionics 

2.  Business 

3.  Unmanned  Ground 

4.  Manned  Space 

5.  Manned  Ground 

6.  Military  Mobile 

7.  Missile  and  Unmanned  Airborne 

8.  Shipboard 

9.  Telecommunications 

10.  Unmanned  Space 

11.  Web 


Comparing  results  from 
an  Avionics  development 
with  that  of  a  business 
project  makes  no  sense 
what-so-ever 


All  comparison  need  to 
take  software  quality 
into  consideration  to  be 
meaningful  (threshold  in 
terms  of  defects/KSLOC 
when  delivered) 
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Application  Domains: 
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Integrity  -  Service  -  Excellence 


|  Application  Domains  | 

1.  Bus 

12.  Platform 

2.  Command  &  Control 

1 3.  Process  Control 

3.  Communications 

14.  Radar 

4.  Controls  &  Displays 

15.  Signal  Processing 

5.  Database 

16.  Simulation  &  Modeling 

6.  Executive 

17.  Situation  Awareness 

7.  Information  Assurance 

18.  Sonar 

8.  Maintenance  &  Diagnostics 

19.  Test  &  Evaluation 

9.  Mission  Management 

20.  Tool  and  Tool  Systems 

10.  Mission  Planning 

21.  Training 

11.  Payload 

22.  Weapons  Delivery 

W 


Software  Sizing: 
Counting  Rules  &  Standards 
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i  Define  software  size  measures  used  as  input  to  cost 
estimation  models,  and  provide  guidelines  for  counting 
and  normalizing  software  size. 

■  Provide  rules  and  guidelines  to  convert  size  inputs  between  models 
so  projects  can  be  represented  in  all  models  in  a  consistent  manner 

■  Logical  source  statements  consisting  of  data  declarations  executables 

■  Rules  for  considering  statement  type,  how  produced,  origin,  build,  etc. 

■  Providing  automated  code  counting  tools  adhering  to  definition  , 
including  initial  modified-code  counting 

■  Providing  conversion  guidelines  for  physical  statements 
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Equivalent  SLOC: 
Stutzke’s  Counting  Rules 
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For  adapted  software,  apply  the  parameters: 

■  DM:  %  of  design  modified 

■  CM:  %  of  code  modified 

■  IM:  %  of  integration  required  compared  to  integrating  new  code 

■  Normal  Adaptation  Adjustment  Factor  AAF  =  0.4*DM  +  0.3*CM  +  0.3*IM 

Reused  software  has  DM  =  CM  =  0. 

■  IM  is  not  applied  to  the  size  of  the  reused  software  (e.g.,  70M  SLOC  of 
Windows  Vista)  but  to  the  size  of  the  other  software  directly  interacting 
with  it  (frequently  estimated  using  a  %) 

Modified  software  has  CM  >  0.  Since  data  indicates  that  the  AAF 

factor  tends  to  underestimate  modification  effort  due  to  added 

software  understanding  effects,  two  other  factors  are  used: 

■  Software  Understandability  (SU):  How  understandable  is  the  software  to 
be  modified? 

■  Unfamiliarity  (UNFM):  How  unfamiliar  with  the  software  to  be  modified  is 
the  person  modifying  it? 
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Step  2:  Indentify  Candidate  Sources  |  \ 


Space,  Ground,  Air 

Defense  Cost  Analysis  Resource  Center 

DD  Form  2630 

340 

Space 

AEHF,  MILSTAR,  GPS 

SEER 

-100 

Air 

F-22  Increment  II 

DD  Form  2630 

13 

Space 

FAB-T 

DD-Form  2630 

24 

Space 

NPOESS 

SEER 

67 

Space,  Air,  Ground 

Northrop  Grumman,  Raytheon 

COCOMO,  SEER 

81 

Air,  Ship,  Ground 

Naval  Center  for  Cost  Analysis 

DD  Form  2630 

68 

Air 

Lockheed  Martin 

COCOMO 

10 

Air 

Army  Cost  and  Economics  Analysis  Center 

DD  Form  2630 

16 

Space 

NRO  CAIG 

SEER 

40-60 

Space 

Aerospace,  Space  &  Missile  System  Center 

SEER 

TBD 

Space 

NASA  JPL 

NASA 

TBD 

Space,  Air,  Ground 

USC  Affiliates 

COCOMO 

TBD 
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Step  3:  Collect  Data 
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USC  will  interview  program  offices  and 
developers  to  obtain  additional  information 
or  resolve  data  anomalies 

1.  SLOC  reporting  -  logical,  physical,  NCSS,  etc. 

2.  Requirements  Volatility  and  Adaptation 

■  Modified  or  Reused  using  DM,  CM,  IM;  SU,  UNFM  as  appropriate 

3.  Size  Type  -  Modified,  Generated,  New,  Re-host,  COTS,  etc. 

4.  Effort  reporting  -  phase  and  activity 

5.  Quality  measures  -  defect  density,  defect  containment,  etc. 

6.  Source  -  in-house,  third  party,  Prior  Build,  Prior  Spiral,  etc. 

7.  Requirements  Volatility  --  %  of  ESLOC  reworked  or  deleted  due 
to  requirements  volatility 

8.  Programming  Languages 

9.  Cost  Model  Parameters  -  True  S,  SEER,  COCOMO 
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Step  4:  Organizational  Taxonomy 
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Productivity  Benchmarks 
(Example  1  -  Not  Endorsed) 


mil 


Operating 

Environment 

•  Similar  systems 

•  Similar  products 

•  Similar  operating 
characteristics 

•  Similar 
requirements 


Application 
Domains 

•  Environment 
independent 

•  Application-oriented 

•  Technology  driven 

•  Characterized 
differently  using 
model  cost  drivers 


Productivity 

Groups 

•  0-25  KSLOC 

•  26-50  KSLOC 

•  51-100  KSLOC 

•  100+  KSLOC 


Meaningful 

Comparisons 


USC  Research  Results 

Productivity  comparisons/benchmarks  show  best  results 
achieved  when  similar  application  domains  in  similar 
operating  environments  are  compared  using  actual  data 
that  is  consistent  and  defendable 


Operating 

Environment 

|  Application  Domain  | 

Projects] 

Size  Range  1 
(EKSLOC)  | 

ESLOC / SM 

LOW 

1  AVE  1 

HIGH 

Avionics 

Command  &  Control 

2 

8 

506 

67 

74 

82 

Communications 

4 

26 

125 

145 

302 

399 

Controls  &  Displays 

3 

733 

746 

183 

241 

312 

Maintenance  &  Diagnostic 

3 

6 

205 

259 

283 

329 

Mission  Management 

18 

1 

1,581 

38 

155 

585 

Mission  Planning 

4 

43 

542 

43 

376 

542 

Radar 

11 

6 

268 

33 

111 

418 

Signal  Processing 

6 

13 

444 

98 

213 

429 

Simulation  &  Modeling 

6 

32 

560 

188 

683 

1552 

Test  &  Evaluation 

1 

21 

21 

34 

34 

34 

Weapons  Delivery 

2 

29 

32 

88 

90 

93 

Source:  Defense  Automated  Cost  Information  System  (DACIMS),  Proprietary  Sources 


USC  Research  Results 

Productivity  comparisons/benchmarks  show  best  results  achieved  when 
similar  application  domains  in  similar  operating  environments  are 
compared  using  actual  data  that  is  consistent  and  defendable 
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Productivity  Benchmarks 
(Example  2  -  Not  Endorsed) 


Productivity  Benchmarks 
(Example  3  -  Not  Endorsed) 
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Operating 

1  Application  Domain 

1  N°'  1 

Size  Range  1 

ESLOC  /  SI 

Environment 

Projects 

(EKSLOC)  | 

LOW 

1  AVE  1 

high 

Manned  Ground 

Command  &  Control 

32 

20 

2486 

22 

296 

831 

Controls  &  Displays 

5 

8 

353 

110 

314 

419 

Executive 

2 

71 

424 

78 

264 

450 

Logistics 

1 

231 

231 

290 

290 

290 

Mission  Planning 

11 

44 

2395 

75 

519 

1766 

Platform 

4 

276 

1517 

88 

129 

161 

Process  Control 

3 

39 

172 

215 

352 

485 

Signal  Processing 

1 

286 

286 

358 

358 

358 

Simulation  &  Modeling 

1 

81 

81 

98 

98 

98 

Situational  Awareness 

2 

20 

1453 

140 

218 

297 

T est  &  Evaluation 

7 

1 

16 

33 

59 

81 

Source:  Defense  Automated  Cost  Information  System  (DACIMS),  Proprietary  Sources 


Operating 

Environment 

|  Application  Domain  | 

Projects] 

Size  Range 
(EKSLOC) 

ESLOC  /  SM 

LOW 

1  AVE  1 

HIGH 

Shipboard 

Command  &  Control 

33 

1  91 

46 

304 

Communications 

8 

0  159 

8 

171 

808 

Controls  &  Displays 

7 

3  41 

33 

60 

78 

Database 

2 

5  5 

83 

129 

176 

Executive 

4 

12  77 

48 

85 

159 

Maintenance  &  Diagnostic 

7 

0  144 

13 

444 

1026 

Mission  Planning 

7 

5  88 

32 

104 

203 

Platform 

15 

2  80 

31 

134 

590 

Radar 

17 

1  46 

4 

66 

265 

Simulation  &  Modeling 

14 

1  81 

17 

78 

244 

Sonar 

1 

2  2 

193 

193 

193 

Test  &  Evaluation 

1 

56  56 

69 

69 

69 

Training 

6 

35  46 

55 

98 

217 

Weapons  Delivery 

9 

4  369 

83 

243 

527 

Source:  Defense  Automated  Cost  Information  System  (DACIMS),  Proprietary  Sources 


•A  similar  table  will  be  provided  for  the  other  8  operating  environments 
•Descriptive  statistics  and  project  descriptions  will  follow  each  table 
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Default  ESLOC  Factors 
(Example  4  -  Not  Endorsed) 
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Size  Type 

Description 

Projects 

ESLOC  Factor 

Low  AVE  High 

Reused 

Pre-existing  code  that  is  not  changed  with  the  adaption  parameter  settings: 

•Design  Modification  %  (DM)  =  0% 

•Code  Modification  %  (CM)  =  0% 

0% 

9% 

40% 

Modified 

(High) 

Pre-existing  code  that  is  changed  with  the  adaption  parameter  setting: 

•Code  Modification  %  (CM)  >  25% 

21 

22% 

51% 

100% 

Modified 

(Low) 

Pre-existing  code  that  is  changed  with  the  adaption  parameter  setting: 

•Code  Modification  %  (CM)  <  25% 

38 

3% 

16% 

34% 

Generated 

Software  created  with  automated  source  code  generators  using  different 
technologies.  It  may  consist  of  the  generator  statements  directly  produced  by 
the  programmer  or  the  3GL  generated  statements  from  automated  tools. 

40 

0% 

6% 

50% 

Re-Host 

Rehosting  software  from  one  target  environment  to  a  similar  environment. 
Assumes  no  major  operating  system  changes.  Development  tools  may  be 
different  between  the  platforms. 

10% 

16% 

25% 

COTS 

Pre-built  commercially  available  software  components  whereby  the  source 
code  is  not  available  to  application  developers.  It  is  not  included  for 
equivalent  size. 

Other  unmodified  software  not  included  in  equivalent  size  are  Government 
Furnished  Software  (GFS),  libraries,  operating  systems  and  utilities. 

1% 

1% 

ESLOC  Factor  =  Adaptation  Adjustment  Factor  =  0.4  x  DM  +  0.3  x  CM  +  0.3  x  IM 
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Raw  Datasets 

(Example  5  -  Not  Endorsed) 


I  I  1  I  I 


(ESLOC/MM)  (SLOC) 


3 


S3 


•In  case  you  want  to  challenge  or  refine  the  benchmarks  and  metrics  provided  in  the  manual, 
the  raw  datasets  (Non  Proprietary  Version)  will  be  included  in  the  appendix 
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■  Already  collected  a  significant  amount  of  data 

■  345  projects  --Defense  Cost  Analysis  Resource  Center 

■  240  projects  -  Raytheon,  Lockheed,  Northrop  Grumman,  etc. 

■  Expecting  space  software  projects  from  National  Reconnaissance 
Office,  NASA,  and  Military  Prime  Contractors  (>100  projects) 

■  Analyzing  over  200  projects 

■  Common  Data  Definitions  and  Standards 

■  Initial  Review  (May  2009) 

■  Interim  Review  (International  COCOMO  Forum,  Oct  2009) _ 


Framework  and  definitions  are  done  as  is  the  initial  data  analysis  - 
detailed  data  analysis  is  in  process  as  is  adding  guidelines 


■  Manual  Publication 

■  Initial  Release  (Sep  2009) 

■  Subsequent  Releases  (Sep  2010,  Sep  201 1 ,  Sep  2012) 
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Concluding  Remarks  \  1 


■  Goal  is  to  publish  a  manual  to  help  analysts  develop 
quick  software  estimates  using  empirical  metrics  from 
recent  programs 

■  Additional  information  is  crucial  for  improving  data 
quality  across  DoD 

■  We  want  your  input  on  Productivity  Domains  and  Data 
Definitions 

■  Looking  for  collaborators 

■  Looking  for  peer-reviewers 

■  Need  more  data 

■  Need  even  more  data 
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Questions  or  Comments? 


■  Any  questions 

■  Any  pointers 

■  Any  feedback 

Contact: 

Wilson  Rosa,  AFCAA 
(703)  604  -0395 
Wilson.Rosa@pentagon.af.mil 
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